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ABSTRACT 


Human  judgement  and  the  process  of  decision  making  have  been 
studied  in  depth  for  the  past  century.  More  recent  research  has 
revealed  that  feedback  is  a  primary  element  in  the  decision  making 
process.  Feedback  has  been  categorized  according  to  its  role  in 
decision  making.  Some  categories  of  feedback  include  cognitive, 
feedforward,  and  outcome.  Cognitive  feedback  may  hold  the  most 
promise  for  positively  affecting  the  decision  making  process. 

Naval  Intelligence  analysis  is  a  complex  process  which  involves 
human  judgement  and  decision  making  on  a  daily  basis.  This  thesis 
sought  to  determine  that  cognitive  feedback  would  enable 
intelligence  analysts  to  make  optimal  choices  more  consistently 
than  if  they  were  presented  with  just  outcome  feedback.  Naval 
Intelligence  analysts  were  the  subjects  of  an  unclassified 
experiment  which  captured  a  realistic  task  performed  routinely  by 
analysts  in  the  Fleet.  The  experiment  revealed  that  cognitive 
feedback  produced  the  most  accurate  and  optimal  results,  and 
indicates  that  intelligence  analysis  would  benefit  from  decision 
support  systems  that  incorporate  the  element  of  cognitive  feedback. 
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I .  INTRODUCTION 


A.  BACKGROUND 

Research  on  decision  making  has  revealed  the  role  of 
feedback  in  affecting  the  decisions  of  individuals  (Hogarth, 
1987).  Feedback  affects  the  choice  that  a  decision  maker  will 
make,  and  can  be  a  crucial  element  in  the  decision  making 
process,  as  it  affects  both  the  task  environment  and  the 
decision  maker's  perception  of  the  environment  (Hogarth, 
1987)  . 

Feedback  can  be  categorized  into  several  different  types, 
including  cognitive  and  outcome  feedback.  As  described  by 
Jacoby ,  et  al ,  1984 , 

...outcome  feedback  is  information  that  describes  the 
accuracy  or  correctness  of  the  response,  cognitive 
feedback  represents  information  regarding  the  how  and  why 
that  underlies  this  accuracy. 

A  type  of  feedback  which  has  shown  beneficial  results  is 
cognitive  feedback  (Balzer  et  al ,  1989).  Cognitive  feedback 
may  have  particular  utility  in  a  complex,  probabilistic 
environment,  where  pertinent  information,  provided  to  the 
decision  maker  in  a  timely  fashion,  can  lead  to  improved 
decisions.  An  example  of  such  an  environment  is  military 
intelligence  analysis.  The  examination  of  the  effects  of 
cognitive  feedback  on  intelligence  analysts  may  lend 
validation  to  current  beliefs  about  cognitive  feedback,  and 
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raise  additional  questions  which/  when  answered,  could  provide 
a  basis  for  future  intelligence  decision  support  systems. 

B.  EXPERIMENTAL  TASK 

To  best  capture  the  effects  of  cognitive  feedback  on 
intelligence  analysis,  a  task  specifically  designed  to  capture 
the  analysis  process  of  analysts  in  the  Navy  was  devised.  The 
task  'ias  presented  to  active  duty  Naval  Intelligence  analysts 
in  a  format  similar  to  what  is  used  in  the  Fleet.  The  entire 
experiment  was  unclassified,  and  the  geopolitical  scenario  was 
fictitious,  yet  effort  was  taken  to  design  a  realistic  task 
environment  that  the  analysts  would  be  familiar  with. 

C.  RESEARCH  QUESTION 

Previous  studies  of  military  intelligence  analysts  have 
revealed  that  analysts  will  tend  to  seek  confirmation  of  their 
decisions  (Tolcott  et  al ,  1989).  If  this  confirmation  can  be 
provided  in  the  form  of  cognitive  feedback,  will  the  analysts 
use  the  feedback  to  reach  more  correct  and  optimal  decisions? 
Does  the  form  of  feedback  matter,  or  will  there  be  a 
noticeable  variance  between  subjects  provided  with  and  without 
cognitive  feedback?  Answers  to  these  questions  were  sought 
through  the  design  and  implementation  of  the  experiment,  and 
were  the  focus  of  this  research. 

Analysts  were  randomly  divided  into  two  groups:  cognitive 
feedback  and  outcome  feedback.  Each  group  was  presented  with 
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the  same  exact  task,  but  the  information  presented  as  feedback 
varied,  dependent  upon  which  group  the  analyst  was  assigned 
to.  The  results  of  each  group  were  compared  to  one  another, 
and  the  effects  of  different  types  of  feedback  on  the  analysts 
decisions  were  collected. 

D.  CONTRIBUTION 

The  importance  of  the  decision  making  process  to 
intelligence  analysis  is  recognized,  and  the  need  to  determine 
which  types  of  feedback  are  optimal  has  been  previously  stated 
(Thompson  et  al ,  1984).  This  research  attempted  to  further 
determine  if  cognitive  feedback  is  relevant  and  beneficial  to 
the  process  of  intelligence  analysis. 

Chapter  II  provides  a  literature  review  of  research  in  the 
area  of  intelligence  analysis,  and  lays  the  theoretical 
premise  for  the  study.  Chapter  III  describes  in  detail  the 
experimental  method.  Chapters  IV  and  V  discuss  statistical 
results  of  the  experiment  and  conclusions  drawn  from  the 
experiment,  respectively.  Results  of  this  study  and  any 
follow-on  studies  may  be  used  to  develop  more  advanced 
decision  support  system  prototypes  for  the  Naval  Intelligence 
and  other  military  intelligence  communities. 
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II.  THEORETICAL  PREMISE 


A.  THE  DECISION  MAKING  PROCESS 
1.  Complexity 

Human  decision  making  is  a  complex  process.  Studies 
of  this  process  have  revealed  that  the  complexity  of  decision 
making  is  dependent  upon  many  individual  factors  such  as 
intuition,  anticipatory  tendencies,  and  judgemental  heuristics 
(Hogarth,  1987).  A  large  portion  of  the  decision  making 
process  involves  human  memory  and  the  capability,  or  lack 
thereof,  of  the  human  mind  to  process  information  (Hogarth 
1979)  . 

The  human  being  is  a  fallible  decision  maker  (Hogarth 
1979).  The  human  mind  is  unable  to  effectively  process 
continuous  information  from  a  multitude  of  sources,  especially 
in  a  stressful  environment.  A  stressful  environment  places 
the  human  mind  in  a  position  to  ignore  some  information  and 
process  other  information,  dependent  upon  existing  heuristics 
(Hogarth,  1987).  In  applications  where  decision  making  is 
vital,  such  as  diagnostic  medicine  and  tactical  military 
scenarios,  it  is  imperative  that  the  information  assigned  the 
least  importance  is  indeed  worthy  of  such  a  low  value. 

Each  piece  of  pertinent  information  contributing  to  a 
decision  must  be  appropriately  presented  to  the  decision  maker 
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in  a  manner  that  will  ensure  the  information  is  processed  with 
due  value.  It  is  beneficial  to  fully  u*.derstand  the 
importance  of  information  presentation  in  the  decision  making 
process.  This  understanding  can  be  accomplished  by  studying 
various  decision  models  as  described  in  subsection  II. A. 2. 

2.  Decision  Making  Models 

Researchers  have  attempted  to  capture  the  human 
decision  making  process  through  observation  and  mathematical 
modeling.  Two  models  of  decision  making  are  the  conceptual 
judgement  model  (Hogarth,  1987)  and  the  lens  model,  as 
described  by  Brunswick  (Libby,  1981).  The  conceptual 
judgement  model  and  the  lens  model  appear  to  approach  the  task 
of  decision  making  from  the  different  aspects  of  psychology 
and  mathematics.  Yet  the  two  models  are  inextricably  linked 
by  the  element  of  feedback,  a  vital  part  of  the  decision 
making  process.  Section  II.C  identifies  and  defines  feedback 
and  its  varieties  in  more  detail.  By  examining  each  of  these 
models  a  point  of  departure  for  this  study  was  attained. 

a.  Judgement  Model 

Hogarth's  conceptual  judgement  model  divides  the 
process  of  decision  making  into  seven  basic  steps:  (1)  task 
environment,  (2)  Schema,  (3)  Acquisition,  (4)  Processing,  (5) 
Output,  (6)  Action,  and  (7)  Outcome.  Step  (7),  outcome, 
provides  information  that  will  be  pertinent  to  future 
decisions.  This  outcome  of  the  decision  making  process  is 
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also  considered  a  form  of  feedback  for  future  judgmental 
scenarios.  Hence  outcome,  or  feedback,  is  a  means  by  which 
future  judgements  can  be  affected.  In  becoming  pertinent 
information  to  the  decision  making  process,  feedback 
establishes  its  vital  role  in  the« decision  making  process. 

The  conceptual  model  of  human  judgement  is 
depicted  in  Figure  1.  Feedback  plays  a  crucial  role  in  the 
judgement  process,  as  it  provides  criteria  to  both  the  task 
environment  and  the  judgement  schema.  The  lens  model  as 
interpreted  by  Brunswick  continues  the  description  of  the 
element  of  feedback  in  mathematical  detail. 

b.  The  Brunswick  Lens  Model 

The  Brunswick  Lens  Model  allows  the  environment  to 
be  observed  through  a  "lens  of  imperfect  cues"  (Libby,  1981), 
and  assigns  weights  to  each  of  the  cues  in  a  specific  decision 
making  scenario.  The  Lens  Model  is  readily  adapted  to 
situations  in  which  the  decision  making  occurs  in  a 
probabilistic  environment,  and  the  accuracy  of  a  decision  is 
contingent  both  on  the  individual  decision  maker  and  the 
environment  in  which  the  decision  is  made  (Hogarth,  1987). 

The  lens  model,  as  depicted  in  Figure  2, 
represents  the  differences  between  actual  value  weights 
assigned  to  pieces  of  pertinent  information  by  the 
environment,  and  the  value  weights  assigned  to  the  pertinent 
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Figure  1.  Conceptual  Judgement  Model  (Hogarth,  1987) 
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pieces  of  information  by  the  decision  maker.  These  pieces  of 
information  which  are  considered  pertinent  to  a  specific 


decision,  or  task,  are  formally  referred  to  as  cues. 


The  lens  model  uses  identical  polynomial  equations 
for  each  side  of  the  model;  the  environment  and  the  decision 
maker.  The  differences  lie  in  the  assigned  values,  or 
weights,  for  each  of  the  cues.  All  of  the  variables  used  by 
the  lens  model  are  listed  and  defined  in  Table  1. 
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TABLE  1.  LEWS  MODEL  VARIABLES  (c.f.  Libby,  1981) 


Symbol 

name 

Definition 

Re 

Envi ronmental 

Pr edi c t abi 1 i t y 

K 

Predictability 

G 

Matching  Index 

r. 

Achievement 

-IZda _ 

First,  the  lens  model  attempts  to  capture  the 
status  of  the  environment.  The  environment  is  represented  by 
equation  1. 

F.*a  *b^x^  + .  .  .  b^^ 


(1) 

Yg,  the  actual  model  imposed  by  the  environment,  is  a 
compilation  of  the  weights  (b|....bQ)  assigned  to  the  cues 
{X|,..Xjj)  by  the  environment.  Likewise,  the  decision  maker's 
view  of  the  task  is  captured  in  an  identical  equation,  Yj , 
with  the  weights  (b^....bj,)  assigned  to  the  cues  (xp..XQ)  by 
the  decision  maker. 

Both  the  environment  and  the  decision  maker  are 
using  the  same  set  of  cues,  (x^...Xq),  to  which  weights  are 
then  assigned.  In  the  most  ideal  situation,  the  decision 
maker  employs  consistent  weights  for  specific  cues,  dependent 
upon  the  nature  of  the  task. 
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other  factors  that  affect  the  decision-making 
process  and  are  thus  represented  in  the  lens  model  include 
consistency  and  environmental  noise.  Environmental  noise  is 
also  referred  to  as  predictability  (Libby,  1981).  Both 
consistency  and  predictability  utilize  similar  variables  in 
their  calculations. 

Rj,  consistency,  measures  how  consistent  the 
decision  maker  is  in  predicting  the  environment.  It  is 
calculated  by  a  multiple  correlation  between  the  cues  and  the 
decision  maker's  judgements. 

Rj,  predictability,  measures  how  predictable  the 
environment  is.  If  noise  or  predictability  is  low,  the 
probability  of  the  decision  maker  attaining  a  reasonable  task 
accuracy  is  low  (Libby,  1981).  Predictability  can  be 
calculated  by  employing  a  multiple  correlation  of  the  cues  and 
the  actual  environmental  values. 

The  accuracy  of  decisions  can  be  measured  through 
the  lens  model,  in  the  form  of  decision  achievement,  rj. 


(2) 

Consistency,  Rj,  and  predictability,  Rj,  can  be  calculated  to 
determine  the  overall  accuracy  of  the  decisions  made  by  the 
decision  maker.  Decision  achievement  is  an  indication  of  how 
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well  an  individual,  Yj,  matches  the  environment,  Y^,  through 
this  "lens  of  imperfect  cues"  (Libby,  1981). 

The  Brunswik  Lens  Model  lends  well  to  the  process 
executed  by  Naval  Intelligence  and  Cryptological  analysts.  By 
examining  the  analyst's  decision  and  mathematically  comparing 
it  to  the  environment's  actual  outcome,  an  index  of 
achievement  for  intelligence  analysts  can  be  arrived  at  for  a 
specific  type  of  analysis  task.  The  effects  of  various 
factors,  such  as  feedback,  on  the  decisionmaking  process,  can 
be  examined  by  comparative  analysis  of  achievement  indices  for 
multiple  instances  of  a  generic  task. 

B.  INTELLIGENCE  ANALYSIS  AS  A  DECISION  MAKING  PROCESS 

While  intelligence  analysis  can  be  a  complex  decision 
making  process  involving  human  judgement,  intelligence 
analysis  has  not  been  "intensively  investigated  by 
psychologists  for  more  than  thirty  years"  in  the  manner  human 
judgement  and  decision  making  have  been  (Tolcott  et  al  1989). 
A  preliminary  literature  review  reveals  somewhat  sporadic 
research  in  the  area  of  intelligence  analysis,  with  the 
majority  of  the  material  focused  on  developing  decision 
support  aids  for  the  intelligence  analyst. 

In  1974  Patten  attempted  to  initiate  a  method  of 
organizing  data  and  information  received  by  an  intelligence 
system  to  help  analysts.  Patten's  concern  was  the  following: 
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[that]  intelligence  analyst[s]  working  on  specific 
problems  must  be  able  to  selectively  obtain  information 
relevant  to  a  specific  problem  without  becoming  bogged 
down  in  irrelevant  information.”  (Patten,  1974) 

Patten's  study  focused  on  the  information  and  data  relevant  to 

the  analysis  process,  but  not  the  analysis  process  itself. 

Sticha,  Patterson,  and  Weiss  researched  and  determined 

Approximate  Reasoning  Methods  for  Decision  Aids  in  1982,  which 

led  to  the  development  of  a  prototype  decision  aid  for  Air 

Force  target  nomination.  The  reason  for  studying  approximate 

reasoning  methods  was  to  hopefully  "...achieve  a  system  that 

could  facilitate  an  intelligence  analyst's  efforts"  (Sticha  et 

al ,  1982).  Hence,  the  actual  intelligence  analysis  process 

became  the  focus  of  the  research. 

An  Intelligence  Aid  for  Estimating  Enemv  Courses  of  Action 
(AI/ENCOA)  was  developed  in  1985  in  an  attempt  to  provide 
cognitive  support  to  intelligence  analysts  by  altering 
decision  rule  bases  (Lehner  et  al  1985).  AI/ENCOA  software 
was  developed  as  a  prototype  decision  aid  to  assist  Army 
tactical  intelligence  analysts  assigned  with  choosing  feasible 
enemy  courses  of  action.  This  prototype  utilized  Multi- 
Attribute  Utility  (MAU)  models  and  enabled  the  analyst  to 
assign  weights  to  each  alternative. 

Another  decision  support  software  package  created  for  Army 
intelligence  analysts  was  DINAA:  The  Dynamic  Intelligence 
Assessment  Aid,  by  Probus  and  Donnell,  in  1986.  DINAA  allowed 
the  analyst  to  determine  probabilities  for  each  avenue  of 
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approach  as  determined  by  AI/ENCOA.  DINAA  utilized  Bayes' 
theorem  to  adjust  and  assess  probabilities.  The  Prototype 
Analyst  Workstation  (PAWS)  developed  in  1990  was  similar  to 
AI/ENCOA  in  that  it  allowed  the  analyst  to  define  alternative 
courses  of  action  open  to  the  enemy,  with  the  aid  of  MAU  (D. 
Thompson  et  al ,  1990). 

Cohen,  Laskey,  and  Tolcott  developed  a  prototype  decision 
aid  in  1987  that  could  be  personalized  for  each  user. 
Submarine  commanders  were  used  as  the  subjects,  and  the  idea 
that 

individual  decision  makers  differ  both  among 
themselves  and  from  task  to  task  in  the 
decision  strategy  they  prefer  (Cohen  et  al , 

1987) 

was  examined.  This  "personalized  and  prescriptive"  decision 
aid  allowed  the  submarine  commander  to  examine  different 
decision  methods,  weights,  and  trade-offs  for  each  decision. 

Tolcott,  Marvin  and  Lehner  conducted  research  in  1989  in 
"...an  attempt  to  further  the  investigation  of  cognitive 
behavior  underlying  intelligence  analysis"  (Tolcott  et  al , 
1989).  This  study  revealed  that  an  intelligence  analyst  will 
tend  to  "[remain}  with  [his/her]  original  estimate"  of  a 
scenario,  regardless  of  update  information  presented  (Tolcott 
et  al ,  1989).  It  appears  as  though  military  intelligence 
analysts  will  "...pay  more  attention  to  supportive  than  to 
contradictory  evidence,"  as  it  pertains  to  their  original 
decision/estimate  (Tolcott  et  al ,  1989).  This  research 
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elicited  another  research  question:  if  update  information 
could  be  defined  as  feedback,  and  if  feedback  could  be  further 
delineated  into  different  types  of  feedback,  would  an 
analyst’s  perception  of  update  information,  or  feedback,  be 
affected  by  the  specific  type  of  feedback  provided?  This 
question  is  addressed  in  section  II. C,  Feedback  and  its  Role 
in  the  Decision  Making  Process. 

1.  Decision  Making  and  Intelligence  Analysis 

The  process  of  intelligence  analysis  is  comparable  to 
the  standard  decision  making  process  but  even  more  pronounced 
due  to  the  high  levels  of  stress,  dynamic  environments,  and 
the  crucial  nature  of  the  decisions  arrived  at.  After 
evaluating  large  amounts  of  incoming  data  (Adelman  et  al , 
1984),  the  analyst  must  "provide  information  on  which 
commanders'  decisions  are  made,"  validating  the  analysts' 
position  as  a  critical  one  in  the  chain  of  intelligence  and 
operational  events  (Lewis  and  Copeland,  1982).  It  is  clearly 
recognized  that  the  analyst  routinely  faces  an  intricate  and 
complex  task  that  may  be  gargantuan  in  its  proportions  (Luckie 
et  al  1968).  Essentially,  the  analyst  is  tasked  with 
repeatedly  "quantifying  assessed  probabilities"  (Probus  and 
Donnell,  1986).  Very  often,  the  analyst  is  relying  solely  on 
intuitive  feelings,  with  little  or  no  substantiated  feedback 
on  his/her  decision  making  process. 
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2.  Intelligence  Analysis  as  a  Process 


Naval  Intelligence  analysis  utilizes  the  decision 

making  process.  Intelligence  analysis  can  therefore  be 

studied  with  the  lens  model  and  conceptual  judgement  model, 

and  rtiay  yield  improved  performance  results  with  the  provision 

of  feedback  to  the  analyst.  Effective  intelligence  analysis 

can  be  defined  as  an  internal ,  concept-driven  process  vice  an 

external,  data-driven  process  (Katter  et  al ,  1979),  and  thus, 

as  an  internal  process,  has  the  needs  of  conceptual  judgement, 

including  feedback  (Hogarth,  1987). 

The  intelligence  analyst  is  continually  faced  with  making 
probability  assessments  concerning  both  categorical  events 
and  events  which  lie  along  a  continuum  (Barclay  and 
Randall,  1975). 

Therefore  the  decision  making  process  used  by  the  analyst 
could  be  ideally  studied  with  the  aid  of  Brunswik's 
interpretation  of  the  lens  model. 


C.  FEEDBACK  AND  ITS  ROLE  IN  THE  DECISION  MAKING  PROCESS 

Technically,  feedback  is  the  process  by  which  an 
environment  returns  to  individuals  a  portion  of  the 
information  in  their  response  output  necessary  to  compare 
their  process  strategy  with  a  representation  of  an  ideal 
strategy  (Balzer  et  al ,  1989) 

Feedback,  as  pictured  simplistical ly  by  Hogarth  in  his 

conceptual  judgement  model ,  is  essential  to  the  decision 

making  process.  Feedback  is  relevant  to  the  decision  making 

process  and  it  is  widely  accepted  that 

Better  performing  individuals  would  be  more  likely  to 
access  and  use  feedback  information  (Jacoby  et  al ,  1984) 
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It  is  fair  to  conclude  from  this  that  feedback  could  play  an 
important  role  in  improving  intelligence  analysis.  There  are 
many  varieties  of  feedback,  such  as  outcome  and  cognitive. 
What  types  of  feedback  will  best  suit  the  process  of 
intelligence  analysis?  The  answer  becomes  clear  when  the  two 
most  basic  types  of  feedback,  outcome  and  cognitive,  are 
examined . 

1.  Outcome  Feedback 

Outcome  feedback  is  the  most  basic  type  of  feedback, 
and  can  be  described  as  the  results  of  a  decision  process  that 
has  already  occurred.  The  results  are  the  outcome(s)  of 
decisions  previously  made;  hence  outcome  feedback  of  a  current 
decision  does  not  facilitate  changing  or  altering  the  current 
decision.  Outcome  feedback  provides  information  on  previous 
decisions  only;  current  decisions  are  unaffected  by  outcome 
feedback  from  their  results.  Outcome  feedback  has  been 
challenged  for  its  validity  as  a  type  of  feedback,  as  it  can 
only  enhance  future  decisions  vice  the  current  decision  making 
process  (Balzer  et  al  1989).  Several  research  studies 
demonstrate  that  outcome  feedback  may  have  little  to  no 
utility  in  an  uncertain  learning  environment  (Balzer  et  al 
1989).  Hence,  outcome  feedback  may  not  be  optimal  in  the 
intelligence  analysis  environment,  as  intelligence  analysis 
frequently  occurs  in  an  uncertain  environment. 
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2.  Cognitive  Feedback 

Cognitive  feedback  provides  the  decision  maker  with 
information  about  the  decision  currently  being  made. 
Cognitive  feedback  allows  the  decision  maker  to  know  what 
their  consistency  rating  is,  what  weights  they  have  assigned 
to  cues,  and  what  weights  the  environment  has  assigned  to  the 
cues.  Cognitive  feedback  consists  of  three  elements;  Task 
Information  (TI),  Cognitive  Information  (Cl),  and  Functional 
Validity  Information  (FVI)  (Balzer  et  al  1989).  Cognitive 
Feedback  provides  what  outcome  feedback  cannot:  information 
on  the  decision  making  process  as  it  occurs.  "Whereas  outcome 
feedback  is  information  that  describes  the  accuracy  or 
correctness  of  the  response,  cognitive  feedback  represents 
information  regarding  the  how  and  why  that  underlies  this 
accuracy"  (Jacoby  et  al ,  1984). 

Naval  Intelligence  analysis  occurs  in  a  very  dynamic 
environment.  Since  it  has  been  suggested  that  outcome 
feedback  is  dysfunctional  in  a  dynamic  environment  (Jacoby  et 
al  1984),  and  that  the  type  of  feedback  believed  to  be  most 
beneficial,  most  often,  is  indeed  cognitive  (Balzer  et  al 
1989),  perhaps  the  feedback  with  the  highest  utility  for  Naval 
intelligence  and  cryptological  analysts  could  be  cognitive 
feedback.  Cognitive  feedback  appears  to  hold  promise  for 
improving  the  quality  of  intelligence  analysis.  This  could  be 
of  the  utmost  importance  since  many  decisions  made  by  analysts 
can  affect  the  safety  and  lives  of  others. 
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D .  HYPOTHESES 


Because  of  the  importance  of  decision  making 
in  intelligence  analysis,  it  is  imperative 
that  more  research  be  devoted  to  the  types  of 
decisions  that  analysts  have  to  make... what 
are  optimum  feedback  mechanisms?  (Thompson  et 
al,  1984) 

This  research  examined  the  following  hypotheses  in  an  effort 

to  answer  the  previous  question: 

Cognitive  feedback  is  more  effective  than  outcome  feedback 
in  intelligence  analysis  tasks. 

Intelligence  analysts  provided  with  cognitive  feedback 
during  their  decision  making  process  will  exhibit  improved 
performance  and  will  make  optimal  vice  satisficing 
decisions . 

The  research  question  was  examined  through  an  experiment, 
using  Brunswik’s  interpretation  of  the  lens  model  and 
mathematical  analysis. 

It  was  imperative  to  first  determine  whether  or  not  the 
subjects  were  affected  by  cognitive  feedback  in  any  way. 
Further  analysis  determined  exactly  how  the  analysts  were 
affected:  positively,  in  that  they  were  making  more  accurate 
and  consistent  decisions;  or  negatively,  in  that  they  were 
making  decisions  which  were  increasingly  inaccurate  and 
inconsistent  with  previous  decisions.  The  effect  of  cognitive 
feedback  on  Naval  Intelligence  and  Cryptological  analysts  was 
the  focus  of  this  study. 
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III.  METHOD 


■t 


A.  INTRODUCTION 

Naval  Intelligence  analysts  make  complex  decisions  on  a 
daily  basis,  whether  their  analysis  is  as  an  individual 
analyst  for  an  operational  command  such  as  a  squadron  or  ship, 
or  they  are  part  of  a  large  watchstanding  team  that 
collectively  analyzes  data  for  the  Department  of  Defense 
(DOD).  Naval  Intelligence  analysis  routinely  involves  complex 
decision  making  and  provides  a  viable  arena  for  examining  the 
effects  of  cognitive  feedback  on  complex  decision  making 
processes . 

The  research  question  was  investigated  in  an  experimental 
setting  designed  to  most  closely  represent  the  type  of 
decisions  made  by  analysts  in  the  Fleet.  A  realistic  and 
fairly  complex  task  of  moderate  to  difficult  complexity  was 
created  in  an  effort  to  employ  the  broadest  range  of  the 
analyst’s  decision  making  capabilities. 

B.  Experimental  Design 

An  experiment  involving  comparison  of  results  between  two 
sets  of  subjects  was  devised.  One  set  of  subjects  (12 
subjects)  was  given  outcome  feedback  only;  another  set  (12 
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subjects)  was  given  cognitive  and  outcome  feedback.  The 
presentation  of  feedback  was  designed  to  provide  an  optimal 
amount  of  information  to  the  subject  with  minimal  complexity 
in  the  presentation  format. 

The  subjects  for  each  part  of  the  experiment  were  randomly 
assigned  to  receive  outcome  or  outcome  and  cognitive  feedback. 
Subjects  receiving  cognitive  feedback  had  a  longer  instruction 
set  to  read  and  understand  before  they  were  able  to  begin  the 
experiment.  Outcome  feedback  was  presented  as  accuracy, 
whereas  cognitive  feedback  was  presented  as  Decision  Support 
Information.  Both  types  of  feedback,  and  how  they  were 
designed  and  presented  to  the  subject,  are  described  in  the 
following  sections. 

C.  PARTICIPANTS 

Subjects  chosen  for  this  experiment  were  all  on  active 
duty  in  the  United  States  Navy.  They  are  currently  serving, 
or  have  served,  in  positions  requiring  intelligence  or 
cryptologic  analysis.  Both  Enlisted  Personnel  and  Commissioned 
Officers  were  used  in  the  experiment.  All  subjects  have 
received  training  by  U.S.  Navy  schools,  and  have  had 
experience  at  making  intelligence  decisions.  The  actual 
positions  and  billets  held  by  the  experiment's  subjects 
varied,  from  operational  intelligence  billets  afloat,  to 
standard  administrative- type  billets  ashore. 
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TABLE  2.  EXPERIMENT  SUBJECT  DEMOGRAPHICS 


Demographic 

Mean 

Standard  Deviation 

Age  (years) 

29.0 

7.0 

Rank  (E-1  through 
0-9) 

9.0  (E-9) 

3.7 

Analysis  Experience 
(years) 

5.5 

4.6 

How  long  since 
conducted  analysis 
in  the  Fleet 
(years) 

1.5 

0.9 

The  subject  demographics  were  calculated  with  the 
assistance  of  Minitab  (Schaefer  and  Anderson,  1989). 


D .  TASK 

The  task  of  "quantifying  assessed  probabilities”  (Probus 
and  Donnell,  1986)  to  various  cues  or  pieces  of  information  is 
central  to  the  intelligence  analyst's  decision  making  process. 
This  is  academically  and  professionally  acknowledged  as  a 
difficult  task,  and  has  been  examined  and  incorporated  into 
the  development  of  different  systems  to  aid  intelligence 
analysis,  such  as  The  Dynamic  Intelligence  Assessment  Aid 
(DINAA)  designed  by  Probus  and  Donnell  in  1986;  and  described 
in  the  User's  Manual  for  the  Prototype  Analyst  Workstation 

( PAWS)  by  Thompson,  et  al ,  in  1990.  Both  DINAA  and  PAWS 

« 

represent  attempts  to  first  capture,  and  then  aid,  the  process 
of  intelligence  analysis.  This  highlights  the  first  challenge 
faced  when  attempting  to  prove  the  hypothesis  of  this  study: 
the  experiment  must  include  a  task  similar  in  complexity  to 
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the  task  performed  by  a  Naval  Intelligence  analyst  on  a  daily 
basis.  The  validity  of  the  experiment  hinged  upon  the  data 
collected  from  the  analysts,  and  the  data  could  only  be  as 
realistic  and  meaningful  as  the  task  itself. 

1.  Selection  of  Task 

The  task  of  intelligence  analysis  requires  the  analyst 
to  provide  a  decision  based  upon  whatever  information  is 
available.  To  represent  this  task  and  ensure  the  robustness 
of  the  experiment,  a  task  similar  to  that  performed  by 
analysts  in  the  Fleet  was  devised.  The  task  requires  the 
analyst,  or  subject,  to  provide  his/her  best  estimate  of  a 
suspect  ship's  position,  based  upon  given  information.  The 
given  information  was  dependent  upon  whether  or  not  the 
subject  had  access  to  solely  outcome  feedback,  or  cognitive 
and  outcome  feedback. 

2.  Description  of  Task 

The  actual  task  presented  to  the  analysts  involved  an 
anti-drug  smuggling  situation.  Anti-drug  smuggling  tasking 
was  chosen  due  to  its  relevancy  in  military  operations  and 
intelligence.  The  task  was  first  presented  to  the  analyst 
from  a  political  standpoint  of  the  situation  in  the  Background 
Information.  The  Background  Information  provided  current 
domestic  and  international  political  and  social  information  as 
it  pertained  to  the  influx  of  illegal  drugs  into  the  United 
States.  The  purpose  of  the  Background  Information  was  to 
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create  a  basis  of  familiarity  for  the  analyst,  and  to  give  the 
task  a  realistic  perspective. 

The  task  was  reduced  to  a  single  situation,  similar  to 
the  analysts  daily  tasks  in  the  Fleet,  in  the  Situation  Report 
and  Scenario,  both  which  narrowed  the  scope  of  the  illegal 
drug  trafficking  trade  to  a  single  suspect  vessel.  The 
analyst  was  asked  to  provide  his/her  best  estimate  of  the 
location  of  the  suspect  vessel  based  upon  the  information 
provided . 

Basic  information  provided  on  the  suspect  vessel 
included  that  it  had  completed  three  previous  runs  from 
Colombia  to  the  U.S.  over  the  past  six  weeks,  tracking  east  of 
Cuba  the  first  time,  and  west  of  Cuba  the  last  two  times.  The 
analyst  was  told  the  vessel  was  estimated  to  carry  up  to  three 
tons  of  cocaine,  and  that  it  was  preparing  to  depart  Colombia 
once  again  for  the  U.S.  mainland.  This  basic  information  on 
the  suspect  vessel  provided  the  analyst  with  a  brief  history 
of  the  vessel,  the  general  geographic  location  of  the 
scenario,  and  an  idea  of  the  vessel's  drug  carrying 
capabi 1 ity . 

3.  Task  Variables/Cues 

A  realistic  task  necessitates  variables  and  random 
weighting  of  the  variables.  This  task  incorporated  four 
variables:  weather,  ship  type/speed,  presence  of  the  Cuban 
Navy,  and  the  last  contact  information  on  the  suspect  vessel. 
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All  four  variables  were  present  throughout  the  task;  thus  the 
analyst  had  information  regarding  each  of  the  four  variables 
each  time  he/she  was  asked  to  perform  the  task  of  providing  an 
estimated  suspect  vessel  position.  Each  variable  appeared  to 
be  weighted  differently  in  each  scenario  presented  to  the 
analyst,  and  the  combination  of  the  differently  weighted 
variables  varied  as  well.  This  was  to  provide  as  realistic  a 
task  as  possible,  and  to  optimally  test  the  analysts  decision 
making  capabilities.  However,  the  actual  position  of  the 
ship,  as  calculated  by  the  system,  weighted  the  ship 
type/speed,  weather,  and  last  contact  variables  equally  and 
included  a  random  error.  The  proximity  of  the  Cuban  Navy 
variable  was  negligible  in  the  calculation  of  the  ship's 
actual  position.  The  values  used  for  each  of  the  variables  as 
presented  to  the  analysts  are  represented  in  Table  3. 


TABLE  3.  EXPERIMENTAL  CUE  CHARACTERISTICS 


Variable 

Values 

Last 

Contact 

1-12  hours 

Ship 

Type/ Speed 

Sailboat  (6-8  knots) 

Yawl  (10-12  knots) 

Cargo  Ship  (19-20  knots) 

Speedboat  (40-t-  knots) 

Sea  State 

1  =  Calm,  winds  up  to  10  knots 

2  =  Breezy,  winds  up  to  20  knots 

3  =  Heavy  swells,  winds  up  to  45 

knots 

4  =  Hurricane  force,  gale  winds 

up  to  80  knots 

Cuban  Navy 
in  Area 

20  nautical  mile  radius 

15  nautical  mile  radius 

10  nautical  mile  radius 

3  nautical  mile  radius 

E.  PRESENTATION 

1.  Task  Description/Inf ormation 

All  subjects,  regardless  of  the  type  of  feedback 
provided  to  them,  received  the  same  basic  information  to 
enhance  the  naturalness  of  the  experiment.  Backgroxind 
Information  and  Situation  Report  provided  geopolitical 
information  on  the  areas  of  concern  for  DOD  and  the  United 
States  Government.  In  this  experiment.  Background  Information 
stated  that  the  focus  of  the  President  is  on  the  anti-drug 
efforts  of  the  U.S.  This  set  the  stage  for  the  scenario  in 
which  the  subject  was  asked  to  provide  a  position  for  the 
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suspect  vessel.  Scenario  provided  information  as  to  what 
billet  the  subject  was  to  place  him/herself  in,  and  for  what 
purpose  they  were  to  provide  an  estimated  position  of  the 
suspect  vessel . 

In  keeping  with  the  experimental  rule  of  "ten 
scenarios  per  cue"  (Balzer  and  Doherty,  1991),  the  analyst  was 
asked  to  provide  an  estimated  position  of  the  suspect  vessel 
forty  separate  times  during  the  experiment.  Each  scenario  was 
independent  of  any  other,  and  incorporated  the  four  variables, 
or  cues,  provided  to  the  subject.  The  four  variables  were 
last  contact,  ship  type/speed,  weather,  and  Cuban  Naval 
exercise  proximity  to  the  location  of  the  suspect  vessel.  The 
actual  values  of  each  variable  (hours,  knots,  sea  state, 
proximity  of  Cuban  Navy  in  nautical  miles)  were  varied  in  each 
scenario.  Thus  each  scenario  used  the  same  four  variables, 
but  in  a  different  combination  of  values  for  each  instance. 

Variables  were  presented  to  the  analysts  in  the  form 
of  message  sets,  which  are  very  familiar  to  intelligence 
personnel.  An  example  of  a  message  set  is  in  the  Appendix. 

Since  the  entire  experimental  scenario  and  background 
information  was  unclassified,  so  were  the  message  sets,  the 
origin  of  each  message,  and  its  destination. 

The  four  variables;  last  contact,  ship  type/speed, 
weather,  and  Cuban  Naval  activity  proximity  to  the  suspect 
vessel,  were  all  presented  in  the  message  set.  The  analyst 
was  presented  with  forty  message  sets  in  total ,  each  formatted 
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identical  to  the  sample  message  set  in  the  Appendix,  but  with 
the  values  of  the  cues,  or  variables,  randomly  varied. 

2.  Geographic  Representation 

The  task  was  designed  to  best  represent  the  tasks 
performed  by  analysts  on  a  daily  basis  in  the  Fleet.  The  task 
asked  the  analyst  for  his/her  best  estimate  of  the  suspect 
vessel's  position,  based  upon  the  information  presented  in  the 
respective  message  set.  In  a  real  situation,  the  analyst 
would  be  asked  to  provide  an  estimated  position  in  degrees 
latitude  and  longitude.  To  provide  a  more  straightforward 
representation  of  the  task  and  also  to  place  necessary  limits 
on  the  pertinent  geographic  area  of  the  task,  a  grid  square 
was  designed  to  cover  the  geographic  area  of  the  experiment. 

The  analyst  was  asked  to  locate  the  suspect  vessel  in 
an  area  in  the  Caribbean,  north  of  Colombia  and  Venezuela. 
The  geographic  area  of  the  task  is  depicted  in  the  Appendix. 
The  grid  square  was  bounded  by  the  following  four  coordinates: 

12N  72W;  18N  72W;  12N  78W;  18N  78W.  The  grid  square  was 
divided  into  thirty-six  smaller  squares,  and  each  square  was 
approximately  sixty  by  sixty  nautical  miles.  Each  square  was 
assigned  a  two-digit,  unique  number.  This  allowed  for  greater 
ease  in  providing  a  position,  and  thus  greater  ease  in 
calculations  involving  the  positions  of  the  suspect  vessel. 
The  grid  square  did  not,  however,  detract  from  the  continuous 
nature  of  the  latitudinal  and  longitudinal  scales  of 
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measurement.  Continuous  values  for  the  position(s)  of  the 
suspect  vessel  were  preserved  with  the  grid  square  as  each 
position  remained  unique  and  numerically  higher  or  lower  than 
any  other  grid  square.  The  grid  square  numbering  maintained 
the  continuity  of  the  number  line. 

What  was  most  realistic  in  the  experimental  setting 
was  that  the  message  sets  presented  multiple  cues  to  the 
subjects  in  a  format  familiar  to  the  subjects.  This  enhanced 
the  viability  of  the  results  captured  by  the  experiment,  as  it 
allowed  the  analysts  to  complete  a  task  as  familiar  as 
possible  to  them.  Experiment  debriefing  revealed  that  nearly 
every  subject  believed  the  experiment  and  task  to  be  highly 
realistic  and  believable. 

F.  SETTING 

Subjects  received  experiment  Background  Information, 
Scenario,  Situation  Report,  Message  Sets,  Computer  Instruction 
Set,  and  Decision  Support  Information  manually,  in  a  loose- 
leaf  binder.  The  experimenter  presented  the  subject  with  the 
binder  after  briefly  explaining  the  geographical  area  of  the 
experiment  with  the  aid  of  a  navigational  chart.  Background 
Information,  Scenario,  Situation  Report,  and  Message  Sets  were 
described  in  sections  III.D  and  III.E.  Computer  Instruction 
Set  and  Decision  Support  Information  will  be  described  in  this 
section,  and  section  III.G,  respectively. 
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1.  Geographic  Area 


A  navigational  chart  identical  to  those  used  in  the 
Fleet  by  analysts  conducting  actual  tasks  was  provided,  with 
the  task  area  delineated  in  grid  format.  Thirty-six  squares, 
approximately  60  nautical  miles  X  60  nautical  miles  each,  were 
numbered  to  represent  a  continuous  scale  similar  to  latitude 
and  longitude.  Each  square  was  unique.  The  grid-square 
numbering  system  allowed  for  greater  ease  in  statistical 
analysis  without  detracting  from  the  actual  continuous 
latitude/longitude  system  employed  in  the  Fleet. 

The  geographic  area  of  the  experiment,  defined  by  the 
grid  square,  was  covered  with  acetate  to  allow  marking  of  the 
area  by  the  analyst  with  a  water  soluble  marker.  Rulers  or 
navigational  plotters/dividers  were  provided  to  the  subject 
for  distance  calculation. 

Each  subject  was  provided  with  a  piece  of  scrap  paper 
and  a  pencil  or  pen  for  any  time-distance  calculations 
required.  Calculators  were  not  permitted  as  a  general  rule. 

2.  System 

The  experiment  was  conducted  on  IBM  compatible  286/386 
computers,  in  private  office  or  computer  laboratory  areas. 
Each  subject  was  given  a  one  page  Computer  Instruction  Set 
which  explained  the  screen  on  which  the  vessel's  estimated 
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position,  in  the  form  of  a  grid  square  number,  would  be 
entered.  Instructions  on  how  to  receive  outcome  or  cognitive 
feedback  were  included  in  the  appropriate  instruction  set. 

Additional  computer  instructions  were  presented  to  the 
subject  on  the  screen  of  the  computer  they  were  using, 
reminding  the  analyst  to  consider  each  message  set 
individually,  and  to  enter  each  two-digit  estimated  position 
of  the  suspect  vessel  into  the  computer  after  they  had 
analyzed  each  message  set.  The  last  instruction  on  the  screen 
directed  the  subject  to  "strike  any  key  when  ready,"  upon 
which  time  the  four-column  screen  for  position  entry  was 
displayed  on  the  monitor. 

3.  Data  Capture/Entry 

The  system  captured  the  subject’s  estimated  positions 
when  the  subject  entered  their  estimated  position  for  each 
message  set.  The  subject  was  presented  with  a  monochrome 
screen  with  four  columns  of  ten  entry  places  each.  To  the 
greatest  extent  possible,  a  graphical  display  was  used  that 
had  been  found  to  be  effective  for  approximate  comparison  of 
quantitative  information  (Brehmer,  1984).  The  entry  places 
for  the  subject's  estimated  positions  were  numbered  1-40,  and 
further  computer  instruction  information  was  presented  on  the 
bottom  line  of  the  screen.  The  columns  for  data  entry 
covered  the  bottom-half  of  the  screen.  The  top-half  of  the 
screen  remained  blank  for  subjects  receiving  outcome  feedback 


30 


only.  When  the  subjects  receiving  solely  outcome  feedback 
pressed  "END,"  they  were  presented  with  the  actual  position  of 
the  the  suspect  vessel  (accuracy,  or  rj, )  in  the  right  hand 
columns  next  to  their  estimated  vessel  position. 

The  subjects  receiving  cognitive  feedback  were  presented 
with  a  menu  after  they  pressed  "END."  The  menu  was  displayed 
in  the  top  half  of  the  screen.  After  choosing  a  cognitive  or 
outcome  feedback  option,  the  subject  participating  in  the 
cognitive  feedback  experiment  was  provided  with  decision  rule 
scales  in  the  top  half  of  the  screen,  or  consistency  and/or 
accuracy  information  in  the  columns  preceding  or  following  the 
estimated  position  entry  column. 

An  experimenter  was  present  for  the  entire  length  of  the 
experiment,  tc  provide  guidance  to  the  subjects  and  ensure 
proper  task  completion  by  the  subjects.  The  subjects  were 
given  no  time  limit  for  completing  the  forty  tasks  involved  in 
the  experiment.  Subjects  took  anywhere  from  50  minutes  to  135 
minutes  to  complete  the  experiment. 

G.  DESIGN  OF  FEEDBACK 

1.  Outcome  Feedback 

In  the  experiment  version  that  incorporated  outcome 
feedback  only,  the  computer  instruction  set  indicated  that  the 
analyst  could  access  outcome  feedback,  in  the  form  of  the 
suspect  ship's  actual  position,  by  pressing  the  "END"  key  on 
the  keyboard.  Once  the  subject  had  accessed  outcome  feedback, 
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he/she  was  unable  to  change  any  positions  he/she  had  already 
entered . 

In  the  experiment  incorporating  both  outcome  and 
cognitive  feedback,  the  analyst  was  able  to  access  their 
accuracy,  or  outcome  feedback,  by  selecting  the  "accuracy" 
menu  choice  on  the  feedback  menu. 

Outcome  feedback  was  presented  in  similar  fashion  to 
the  subjects  for  both  sets  of  the  experiment.  When  the 
subject  accessed  "accuracy,"  the  outcome  feedback  was 
displayed,  in  the  form  of  a  two-digit  suspect  vessel  position, 
as  estimated  by  the  system.  The  system's  estimate  of  the 
suspect  vessel's  position  appeared  in  the  right-hand  column  of 
the  data-entry  screen,  next  to  the  suspect  vessel's  position 
estimated  by  the  subject.  The  outcome  feedback,  in  the  format 
of  "accuracy,"  is  displayed  in  Figure  3. 

Outcome  feedback,  or  accuracy,  was  calculated  with  the 
use  of  consistency  and  predictability  information. 

2.  Cognitive  Feedback 

All  subjects  conducting  the  outcome  and  cognitive 
version  of  the  experiment  received  an  additional  instruction 
set,  Decision  Support  Information,  which  detailed  the  five 
choices  of  cognitive,  or  "decision"  feedback  available  to  the 
analyst . 
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To  access  both  cognitive  and  outcome  feedback,  the 
analyst  pressed  the  "END"  key,  which  then  provided  them  with 
a  menu  of  information  feedback  options.  Choices  numbered  1 
through  5  on  the  menu  provided  cognitive  feedback  information. 
Choice  number  6  provided  outcome  feedback,  in  the  form  of 
accuracy,  as  described  in  the  previous  subsection. 

The  experimental  model  required  that  the  analyst  had 
provided  positions  for  at  least  15  message  sets  before  any  of 
the  cognitive  feedback  options  could  be  accessed.  Cognitive 
feedback  information  was  presented  to  the  subject  in  the 
following  formats: 

1.  Decision  Rule  Information:  This  provided  specific  weights 
assigned  to  the  variables  by  the  analyst  for  the  previous 
scenarios  which  the  analyst  had  provided  an  estimated 
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position  for.  This  function  would  reveal  to  the  subject 
which  of  the  variables  they  were  placing  the  most  and 
least  values  on  to  make  their  decision.  The  calculation 
of  the  weights  assigned  to  the  variables  was  accomplished 
by  the  following  method  (as  in  Sengupta,  1990): 

(a)  Weights  assigned  to  the  variables  were  calculated  by 
a  multiple  regression  of  variable  values  and  the  analyst's 
estimation  of  the  weights. 

(b)  The  weights  were  then  transformed  into  a  percentage 
representation  of  the  total  value  of  the  weights. 

(c)  The  weights,  transformed  into  percentages,  were  then 
displayed  on  a  horizontal  bar  graph  as  shown  in  Figure  4. 


2.  Consistency  Information:  This  function  provided  the 

subject  with  the  positions  they  would  have  chosen  for  the 
suspect  vessel  if  they  were  indeed  applying  their  decision 
rule  consistently.  As  with  choice  one,  Decision  Rule 
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Information,  only  the  message  sets  for  which  the  analysts 
had  provided  an  estimated  position  could  be  given 
correlated  consistency  positions.  Consistency  was 
calculated  by  a  multiple  correlation  between  the  variables 
and  the  analyst's  estimation  of  the  weights  of  the 
variables.  Consistency  was  presented  to  the  analyst  on 
the  data  entry  menu,  in  the  left-hand  column  next  to  the 
analyst's  estimated  suspect  vessel  positions.  An  example 
of  the  presentation  of  consistency  is  depicted  in  Figure 
5. 


3.  Information  on  the  System’s  Decision  Rule:  This  function 
provided  the  subject  with  the  actual  weights  for  each 
variable,  or  cue,  as  applied  by  the  system,  to  achieve  the 
correct,  or  system,  answer.  This  was  calculated  in  a 
manner  similar  to  Decision  Support  Information,  but  the 
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multiple  regression  was  between  the  actual  weights  of  the 
variables,  and  the  assigned  values  of  the  variables. 
Information  on  the  system's  decision  rule  was  represented 
on  a  horizontal  bar  graph. 

4.  Information  on  the  System's  and  the  Subject's  Decision 

Rule:  a  combination  of  choices  one  and  three,  this 

function  allowed  the  subject  to  graphically  compare 
his/her  applied  decision  weights  with  the  weights  utilized 
by  the  system. 

5.  Information  on  the  Subject's  Decision  Rule  and 

Consistency:  a  combination  of  choices  one  and  two,  this 

function  presented  the  subject  with  both  the  weights 
he/she  was  utilizing,  as  well  as  his/her  degree  of 
consistency . 

6.  Accuracy:  this  function  served  as  the  outcome  portion  of 
this  version's  feedback,  as  it  provided  the  subject  with 
the  actual  vessel  position.  As  in  the  outcome  feedback 
only  version  of  the  experiment,  once  the  subject  accessed 
accuracy,  or  outcome  feedback,  he/she  was  not  able  to 
revise  any  previously  entered  vessel  positions. 
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IV .  EXPERIMEHTAL  RESULTS 


K.  TASK  RESULTS 

The  statistical  analysis  of  the  experiment  supports  the 
hypotheses  by  revealing  that  the  analysts  presented  with 
cognitive  feedback  and  outcome  feedback  performed 
significantly  better  than  the  analysts  who  received  outcome 
feedback  only.  Hence,  cognitive  feedback  impacted  the 
intelligence  analysis  process  pertinent  to  the  experimental 
task  in  a  positive  manner. 

An  immediate  and  clear  measure  of  the  performance  of  the 
analysts  is  represented  in  Table  4. 


TABLE  4 .  MEANS 

AND  (STANDARD  DEVIATIONS)  OF  PEf 

tFORMANCE 

TYPE  OF 

FEEDBACK 

Rs 

6 

Cognitive 

Feedback 

.841  (.106) 

.914  (.068) 

.967  (.041) 

Outcome 

Feedback 

.658  (.192) 

.811  (.077) 

.717  (.180) 

The  measures  of  performance  are  the  achievement  index,  or 
accuracy,  Tj,  the  consistency  index,  Rj,  and  the  matching 
index,  G.  In  all  three  measures,  the  analysts  receiving 
cognitive  feedback  performed  markedly  better  than  the  analysts 
receiving  outcome  feedback  only. 
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To  more  closely  examine  these  results,  an  analysis  of 
variance  (ANOVA)  amongst  the  performance  factors  was 
conducted.  This  was  calculated  using  the  General  Linear 
Models  procedure  in  SAS  (SAS,  1987).  The  results  are  shown  in 
Table  5. 


TABLE  5.  AMOVA  OF  MODEL  AMD  (ERROR) 


Dependent 

Variables 

Degrees 

of 

Freedom 

Sum  of 
Squares 

Mean 

Square 

F  Value 

Pr  >  F 

1 

(22) 

4.720 

(1.904) 

0.079 

(0.019) 

4.05 

0.0001 

Rs 

1 

(22) 

0.624 

(0.398) 

0.010 

(0.004) 

2.56 

0.0001 

G 

1 

(22) 

2.606 

(1.968) 

0.043 

(0.020) 

2.16 

0.0003 

The  results  indicate  that  subjects  receiving  cognitive 
feedback  had  a  significantly  higher  achievement  score  than 
those  receiving  outcome  feedback  (F{1,22)  =  4.05;  p  <  0.0001). 
Subjects  receiving  cognitive  feedback  had  a  significantly 
higher  consistency  score  than  those  receiving  outcome  feedback 
only  (F(l,22)  =  2.56;  p  <  0.0001).  Subjects  receiving 
cognitive  feedback  also  had  a  significantly  higher  matching 
index  than  those  receiving  outcome  feedback  only  (F(l,22)  = 
2.16;  p  <  0.0003).  We  thus  conclude  that  subjects  receiving 
cognitive  feedback  performed  better  than  those  receiving 
outcome  feedback  only.  The  null  hypothesis  is,  therefore, 
rejected.  The  statistical  results  of  the  ANOVA  of  the 
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performance  indicators  also  support  the  hypothesis  that 
cognitive  feedback  positively  affects  the  performance  of 
intelligence  analysts. 

B.  DEBRIEF  RESULTS 

The  debriefing  questionnaires  and  task  block  surveys 
revealed  information  on  the  subjects  perceptions  of  the 
experiment.  Overall,  the  analysts  found  the  task  to  be 
difficult,  and  felt  that  it  moderately  resembled  a  task  they 
had  previously  performed  in  the  Fleet.  Subjects  found  the 
task  to  be  clear,  and  took  the  experiment  very  seriously. 

Analysts  who  received  cognitive  feedback  found  it  to  be 
moderately  helpful,  and  all  subjects  felt  that  the  outcome 
feedback  (actual  ship's  position)  was  moderately  helpful. 
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V .  CONCLOSIOMS 


A.  SUMMARY 

The  purpose  of  this  study  was  to  evaluate  the  efficacy  of 
cognitive  feedback  in  improving  intelligence  analysts' 
performance.  The  statistical  results  of  the  experiment 
indicate  that  analysts  who  received  cognitive  feedback  did 
indeed  perform  better  than  their  colleagues  who  received  only 
outcome  feedback. 

B.  CONTRIBUTION 

Evidence  thus  indicates  that  cognitive  feedback  positively 
impacts  performance.  The  significance  of  this  study  is  that 
it  involved  U.S.  Naval  Intel  1 igence  and  Cryptol ogic  personnel, 
and  indicates  that  cognitive  feedback  could  be  beneficial  to 
more  types  of  intelligence  analysis.  For  example, 
intelligence  analysts  tracking  a  suspect  vessel  would  be  able 
to  receive  information  on  the  significance  of  the  cues  in  the 
particular  environment  in  which  they  are  working 
(environmental  weights).  Analysts  could  have,  at  a  glance,  a 
summary  of  the  emphasis  that  they  have  been  placing  on  various 
cues  in  a  particular  problem  (decision  weights).  Analysts 
could  also  access  information  relating  how  consistent  their 
choices  are  (consistency).  One  or  all  of  these  added  tools  to 
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the  environment  of  intelligence  analysis  could  be  beneficial 
and  lead  to  improved  analysis  results. 

C.  FUTURE  RESEARCH 

Future  research  could  use  this  study  as  a  point  of 
departure  to  determine  which  forms  of  cognitive  feedback  are 
most  beneficial  to  intelligence  analysis.  This  could  lead  to 
the  development  of  a  Decision  Support  System  prototype  for 
U.S.  Naval  Intelligence  activities. 
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APPENDIX 


NAVAL  INTELLIGENCE  OFFICERS:  A  LENS  VIEW  OF  INTELLIGENCE 
ANALYSIS 


INSTRUCTION  SET 


A.  Introduction 

1.  The  purpose  of  this  study  is  do  examine  how  Naval 
Intelligence  Officers  and  Specialists  process  complex 
information . 

2.  Your  participation  in  this  experiment  is  voluntary  and 
the  results  of  your  participation  will  be  used  for  research 
purposes  only.  Please  take  the  experiment  seriously  and 
attempt  to  answer  the  queries  as  realistically  as  possible. 

B.  Your  Task 

1.  Your  task  involves  three  basic  steps.  They  are: 

a.  Read  the  background  information  for  your  scenario 
provided  in  your  task  booklet.  You  may  need  to  read  it  over 
two  or  three  times  before  you  feel  thoroughly  comfortable  with 
it.  The  scenario  will  provide  information  regarding  the  task 
you  are  asked  to  perform.  For  example,  you  will  be  told  the 
history  of  a  particular  drug-smuggling  ship  that  you  will 
later  be  asked  to  provided  an  estimated  location  for. 

b.  Examine,  individually,  each  of  the  40  message  packages 
provided.  Each  message  set  is  on  a  separate  piece  of  paper 
and  consists  of  three  "messages"  similar  to  those  you  use  in 
the  Fleet.  The  scenario  will  take  place  in  the  Caribbean. 
Each  message  set  will  provide  information  on  the  ship's 
type/speed,  the  last  contact  (in  hours),  weather  in  the 
Caribbean,  and  Cuban  Naval  presence  in  the  vicinity  of  the 
suspect  vessel . 

A  typical  message  set  is  composed  of  three  messages 
whose  bodies  of  information  appear  as  follows: 

"SUSPECT  SHIP  LOCATED  15  HOURS  AGO  IN  THE  VICINITY  OF  GRID 
SQUARE  #  373,  TRAVELING  AT  3  XTS.” 

"WEATHER  REPORT  FOR  15  JANUARY:  CLEAR  SKIES,  WINDS  FROM 
THE  WEST  AT  10-12  KTS,  NO  WHITECAPS,  MINIMAL  WAVES." 
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CUBAN  NAVAL  UNITS  SIGHTED  WITHIN  1  NM  OF  SUSPECT  VESSEL." 


c.  After  you  read  and  analyze  each  message  set,  you  are 
to  provide  a  grid  square  position  number  of  where  you  best 
estimate  the  target  unit  to  be  located.  You  will  use  the 
chart  provided  and  the  navigation  set  to  determine  this. 


REMEMBER- -each  message  set  is  independent  of  all  others, 
hence  you  will  estimate  a  new  position  for  each  message  set. 

You  will  complete  two  blocks  of  40  queries,  for  a  total  of 
80  position  estimates. 


d.  This  is  a  not  a  simple  task  and  the  answer  may  not 
seem  "easy"  to  arrive  at.  Do  not  get  discouraged,  and 
remember  that  you  were  chosen  for  this  experiment  because  of 
your  expertise  at  decision-making  in  a  complex  environment. 

2.  Please  read  the  computer  instructions  carefully.  You  will 
receive  the  scenario  background  information  and  message 
packages  manually,  and  enter  your  estimated  position  into  the 
computer.  You  may  also  be  able  to  use  the  information  on  the 
computer  to  receive  feedback  on  your  decisions,  and  this  will 
be  specified  in  the  computer  instructions. 

3.  After  you  complete  the  entire  task  (both  blocks),  please 
fill  out  the  debrief  questionnaire  and  the  demographic 
information  sheet  provided.  Please  be  honest  and  as  clear  as 
possible  in  your  responses.  Turn  your  entire  packet  into  the 
experimenter . 
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Cognitive  Feedback 


COMPUTER  INSTRUCTION  SET 

You  will  be  presented  with  four  columns  on  your  screen. 
There  are  ten  rows  in  each  column,  one  for  each  message  set, 
for  a  total  of  forty  message  sets  per  block. 

*  The  message  sets  will  be  presented  manually.  Each  set  is  on 
a  separate  piece  of  paper.  The  message  sets  incorporate 
information  that  will  enable  you  to  estimate  the  target  ship’s 
location.  Specifically,  you  will  be  given  information  on  the 
ship  type/speed,  last  contact  (in  hours),  weather,  and  Cuban 
Naval  presence  in  the  area. 

*  You  can  enter  your  grid  square  number  choice  for  each 
message  set  next  to  the  appropriate  message  set  number.  You 
need  not  analyze  the  message  sets  in  any  particular  order. 
You  can  maneuver  about  the  screen  with  the  cursor.  The  "up" 
and  "down"  arrows  allow  you  to  select  the  message  sets  you 
choose  to  analyze,  and  also  allow  you  to  change  answers  if  you 
choose  to. 

*  At  anytime  during  the  experiment  you  may  access  decision 
support  information  on  the  accuracy  your  decision  making 
process.  Simply  press  "END"  and  a  decision  support 
information  menu  will  be  presented  to  you  at  the  top  portion 
of  your  screen.  You  may  select  up  to  six  different  types  of 
feedback,  but  once  you  chose  feedback  that  displays  the 
accuracy  of  your  decisions,  you  may  not  change  any  decisions 
already  completed.  You  will  receive  further  information  on 
the  decision  support  information  and  how  to  use  it  before  you 
begin  the  experiment. 

*  All  decision  rule  information  requires  that  you  have 
entered  at  least  10  positions  before  you  can  access  this 
information. 

*  After  you  have  completed  Block  One,  complete  the  Block  One 
Questionnaire  and  press  "HOME"  to  move  to  Block  Two. 

*  After  Block  Two,  complete  the  Block  Two  questionnaire,  the 
task  debrief  survey,  the  demographic  information  sheet,  and 
inform  the  experimenter  that  you  are  finished. 

*  The  line  at  the  bottom  of  the  screen  will  provide  you  with 
information  on  how  to  maneuver  about  the  screen  and  how  to 
receive  additional  information. 

*  IF  AT  ANYTIME  DURING  THE  EXPERIMENT  YOU  ARE  NOT  SURE  OF  HOW 
TO  USE  THE  PROGRAM,  ASK  THE  EXPERIMENTER. 
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Outcome  Feedback 

COMPUTER  INSTRUCTION  SET 

*  You  will  be  presented  with  four  columns  on  your  screen. 
There  are  ten  rows  in  each  colvunn,  one  for  each  message  set, 
for  a  total  of  forty  message  sets  per  block. 

*  The  message  sets  will  be  presented  manually.  Each  set  is  on 
a  separate  piece  of  paper.  The  message  sets  incorporate 
information  that  will  enable  you  to  estimate  the  target  ship's 
location.  Specifically,  you  will  be  given  information  on  the 
ship  type/speed,  last  contact  (in  hours),  weather,  and  Cuban 
Naval  presence  in  the  area. 

*  You  can  enter  your  grid  square  number  choice  for  each 
message  set  next  to  the  appropriate  message  set  number.  You 
need  not  analyze  the  message  sets  in  any  particular  order. 
You  can  maneuver  about  the  screen  with  the  cursor.  The  "up" 
and  "down"  arrows  allow  you  to  select  the  message  sets  you 
choose  to  analyze,  and  also  allow  you  to  change  answers  if  you 
choose  to. 

*  At  anytime  during  the  experiment  you  may  find  out  what  the 
correct  answers  were  for  message  sets  you  have  already 
completed.  By  pressing  "END"  you  will  be  provided  with  the 
"correct"  ship  positions.  Once  you  access  this  information, 
you  may  not  change  any  answers  you  have  already  entered. 

*  After  you  have  completed  Block  One,  complete  the  Block  One 
questionnaire  and  press  HOME"  to  move  to  Block  Two. 

*  After  Block  Two,  complete  the  Block  Two  questionnaire,  the 
task  debrief  survey,  the  demographic  information  sheet,  and 
inform  the  experimenter  that  you  are  finished. 

*  The  line  at  the  bottom  of  the  screen  will  provide  you  with 
information  on  how  to  maneuver  about  the  screen  and  how  to 
receive  additional  information. 

*  IF  AT  ANYTIME  DURING  THE  EXPERIMENT  YOU  ARE  NOT  SURE  OF  HOW 
TO  USE  THE  PROGRT^,  ASK  THE  EXPERIMENTER. 
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BLOCK  ONE 


Background  Information 

The  United  States  has  been  fighting  the  drug  war  for  over 
three  years,  yet  the  effectiveness  of  the  war  is  questionable. 
The  Presidential  election  is  only  one  year  away,  and  the 
current  administration  wishes  to  ensure  continual  funding  for 
the  anti-drug  effort. 

World  politics  have  taken  a  backseat  to  domestic  issues. 
The  current  administration  realizes  that  domestic  policy  has 
not  been  its  strong  suit  in  the  press,  as  presen*-ed  to  the 
American  people.  One  issue  that  is  repeatedly 
spotlighted  by  the  media  is  that  of  the  "crack  babies"  being 
born  to  cocaine  addicted  mothers,  at  an  alarming  rate.  Crack 
babies  are  not  just  born  in  the  inner-city  ghetto  atmosphere, 
but  to  mothers  of  all  social  strata  in  the  U.S.  Recent  news 
commentaries  have  revealed  that  the  importation  of  cocaine  and 
crack  into  the  U.S.  is  at  an  all  time  high.  Congress  is 
beginning  to  ask  what  is  going  on  with  the  drug  war. 

Senior  military  officials  know  that  their  anti-drug 
campaign  has  been  relatively  successful  and  fruitful.  In 
light  of  the  small  number  of  U.S.  armed  forces  dedicated  to 
the  anti-drug  campaign,  the  total  amount  of  captured 
contra-band  is  phenomenal .  But  in  comparison  to  the  total 
amount  of  cocaine  and  crack  smuggled  into  the  U.S.  annually, 
it  is  barely  the  tip  of  the  iceberg. 

The  President  saw  a  need  to  step  up  the  intensity  of  the 
drug  war.  He  elicited  advice  from  his  top  military  personnel 
regarding  just  how  much  military  hardware  and  personnel  would 
be  needed  to  accomplish  this.  Then,  the  President  went  to 
Congress  with  the  Secretaries  of  Defense  and  Transportation 
and  presented  a  proposal  to  attempt  to  finally  put  a  noose  on 
the  cocaine  flow  from  Colombia,  Bolivia,  and  Peru.  The 
country  of  chief  concern  is  Colombia,  since  drugs  harvested  in 
Peru  and  Bolivia  are  transported  through  Colvimbia  and  then 
flown  or  shipped  to  the  U.S. 

Situation  Report  10  November  1991 

In  a  sweeping  majority  last  week.  Congress  voted  to 
increase  the  budget  for  the  drug  war  ten-fold.  This  increase 
was  voted  in  under  the  conception  cf  the  American  people  that 
the  war  could  now  be  tackled  in  a  short  period  of  time  and  put 
an  end  to  the  drug  trade  in  the  Caribbean. 

Scenario 
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You  are  on  watch  in  the  Joint  Team  Against  Drugs  Command 
in  Florida.  Pressure  has  been  increasing  for  your  watch  team 
to  get  the  best  intelligence  possible  out  to  the  Fleet  surface 
and  air  units  who  are  responsible  for  patrolling  marked  areas 
in  the  Caribbean. 

You  have  received  intelligence  indicating  that  a  vessel 
capable  of  carrying  three  tons  of  cocaine  is  preparing  to 
depart  Colombia  for  the  U.S.  mainland.  A  history  of  this 
vessel  reveals  that  it  has  completed  three  runs  from  Colombia 
to  the  U.S.  in  the  last  six  weeks.  Its  tracks  were  around  the 
eastern  edge  of  Cuba  the  first  time,  then  around  the  western 
edge  the  last  two  times.  Track  navigation  took  4  days  each 
time . 

You  are  directed  to  provide  the  most  likely  area  the 
vessel  is  at  the  current  time.  Taking  into  consideration  the 
information  provided  in  the  most  recent  message  traffic,  in 
what  area  would  you  send  the  aircrews  and  ships?  It  is 
currently  11  Nov  and  you  need  to  estimate  the  vessel's 
position.  Please  provide  a  grid  square  number  where  you  best 
estimate  the  suspect  ship  to  be. 


% 
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Decision  Support  Information 


*  What  is  Decision  Support  Information? 

Decision  Support  Information  is  diagnostic  information 
provided  by  the  system  on  your  decision  processes  and  that  of 
the  system.  You  can  access  the  decision  support  when  making 
your  decisions  on  the  suspect  vessel's  positions.  (In  order 
to  compute  the  information,  however,  the  system  needs  at  least 
10  scores  from  you).  By  accessing  such  information,  decision 
makers  can  derive  better  insight  into  their  decisions 
processes.  This  enables  them  to  revise  and  improve  their 
decisions  (or  positions,  in  this  case)  through  a  "what-if" 
mode  of  analysis. 


*  How  do  I  use  Decision  Support  Information  in  Making 
Decisions? 

Typically,  you  the  decision  maker,  would  use  the  decision 
support  information  as  follows: 

1.  Make  some  tentative  decisions  (i.e.,  ship's 
positions) . 

2.  Ask  the  system  for  decision  support  information. 

3.  Refine  your  positions  accordingly. 

You  may  wish  to  do  this  1-3  sequence  within  a  block,  as 
many  times  (and  with  as  many  scores)  as  you  wish. 

This  sequence  allows  you  to  delay  asking  for  your  accuracy 
and  enables  you  to  continue  to  refine  your  positions. 


*  Types  of  Decision  Support  Information 

(For  illustrations  of  Decision  Support  Information,  see 
III.G.2) 

The  system  will  provide  you  with  the  following  5  types  of 
decision  support  information: 

1.  INFORMATION  ON  YOUR  DECISION  RULE: 

Decision  makers  are  sometimes  unable  to  specify 
precisely,  a  particular  decision  rule  (in  this  case,  weights 
assigned  to  specific  information  regarding  the  suspect 
vessel).  The  system  will  track  the  weights  you  are  using  (in 
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formulating  your  positions)  and  will  display  them  through  a 
stacked-bar  chart. 

How  do  I  use  it? 

1.  Make  sure  the  weights  displayed  are  actually  the 
ones  you  want  applied. 

2.  If  not,  revise  your  scores,  and  see  how  the 
weights  change. 

3.  Iterate  between  1-2  until  the  system  shows 
weights  you  actually  want  applied. 


2.  INFORMATION  ON  YOUR  CONSISTENCY: 

Sometimes,  after  decision  makers  have  specified  their 
decision  rules,  they  are  unable  to  apply  them  consistently. 
The  system  will  calculate  the  scores  you  would  have  given  had 
you  been  completely  consistent  with  your  decision  rule. 

How  do  I  use  it? 

1.  Check  year  scores  against  the  consistency  scores. 

2.  Revise  your  scores  if  you  need  or  wish  to. 

3.  Iterate  between  1-2  till  your  scores  match  with  or 
are  close  to  the  consistency  scores. 


3.  INFORMATION  ON  THE  SYSTEM'S  DECISION  RULE: 

Instead  of  trying  to  figure  out  from  several  examples  what 
rule  the  system  is  using,  it  is  more  effective  if  it  is 
displayed  in  a  bar  graph  format. 

How  do  I  use  it? 

1.  Use  the  information  to  get  an  idea  of  what  decision 
rule  the  system  has  been  following. 

4.  INFORMATION  ON  THE  SYSTEM'S  DECISION  RULE  AND  YOURS 

This  is  actually  a  combination  of  decision  support 
information  1  and  3.  It  enables  you  to  compare  your  decision 
rule  with  that  of  the  system  and  thereby  emulate  the  system 
better . 
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How  do  I  use  it? 


1.  Check  weights  you  have  given  versus  weights  given  by 
the  system. 

2.  Revise  your  positions  if  you  need  to. 

3.  Iterate  between  1-2  until  your  weights  match  with  or 
are  close  to  the  system's  weights. 


5.  INFORMATION  ON  YOUR  DECISION  RULE  AND  CONSISTENCY 

This  is  actually  a  combination  of  decision  support 
information  1  and  2.  The  idea  here  is  to  let  you  revise  your 
weights  without  losing  your  consistency  at  the  same  time. 

How  do  I  use  it? 

1.  Make  sure  the  weights  displayed  are  actually  the  ones 
you  want  applied. 

2.  If  not,  revise  your  positions,  and  see  how  the  weights 
change . 

3.  Check  your  scores  against  the  consistency  scores. 

4.  Revise  your  scores  if  you  need  or  wish  to. 


***YOU  ARE  NOW  READY  TO  PROCEED  WITH  THE  TASK*** 
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MESSAGE  SET  ONE 


PM:  USS  SPIKE 

TO :  JTAD 

SUBJ:  VESSEL  SIGHTING 

SUSPECT  VESSEL  SIGHTED  BY  FANTAIL  LOOKOUT  APPROX.  7  HRS.  AGO, 
AT 

GRID  SQUARE  #68,  MOVING  18  KTS. 


FM :  NOAA 

TO :  JTAD 

USCOMSOLANT 

SUBJ:  WX  REPORT  FOR  11  NOV  1991 

PARTLY  CLOUDY  SKIES  WITH  WINDS  FROM  THE  NNW  AT  UP  TO  45  KTS, 
CAUSING  HEAVY  SWELLS  AND  WHITECAPS. 


FM:  USS  SCUTTLEBUTT 

TO :  JTAD 

USCOMSOLANT 

SUBJ:  CUBAN  NAVAL  EXERCISE  ACTIVITY 

CUBAN  NAVAL  EXERCISE  UNITS  SIGHTED  WITHIN  20  NM  OF  SUSPECT 
VESSEL. 
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QUESTIONS  TO  BE  ANSWERED  AFTER  COMPLETING  BLOCK  ONE 


cognitive  feedback 


1.  Describe  (in  words  or  equations)  what  decision  rule  you 
followed  in  making  your  own  estimates: 


2.  Distribute  100  points  among  the  three  variable  you  used 
for  reaching  your  overall  estimate-in  accordance  with  the 
importance  you  assigned  them  (total  will  add  up  to  100). 

Ship's  speed  _ 

Ship's  last  contact  _ 

Weather  _ 

Cuban  Naval  Presence _ 

100  Total 

3.  What  do  you  think  the  actual  weights  the  system  used  for 
each  of  the  variables  was? 

Ship's  speed  _ 

Ship's  last  contact  _ 

Weather  _ 

Cuban  Naval  Presence _ 

100  Total 

4.  In  this  task,  did  you  request  decision  support  information 
at  any  time  from  the  system?  Y  N 

5.  If  "YES",  try  to  describe  how  you  used  decision  support 
information  in  making  your  decisions. 
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QUESTIONS  TO  BE  ANSWERED  AFTER  COMPLETING  BLOCK  ONE 
outcome  feedback 

1.  Describe  (in  words  or  equations)  what  decision  rule  you 
followed  in  making  your  own  estimates: 


2.  Distribute  100  points  among  the  three  variable  you  used 
for  reaching  your  overall  estimate-in  accordance  with  the 
importance  you  assigned  them  (total  will  add  up  to  100). 

Ship’s  speed  _ 

Ship's  last  contact  _ 

Weather  _ 

Cuban  Naval  Presence _ 

100  Total 

3.  What  do  you  think  the  actual  weights  the  system  used  for 
each  of  the  variables  was? 

Ship's  speed  _ 

Ship's  last  contact  _ 

Weather  _ 

Cuban  Naval  Presence _ 

100  Total 
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DEMOGRAPHIC  SURVEY 


NAME _ AGE _ SEX 

RANK _ DES I GNATOR _ 

COMMISSIONING  DATE _ COMMISSIONING 

SOURCE _ 

YEARS  IN  INTELLIGENCE  FIELD _ 

PREVIOUS  OPINTEL  EXPERIENCE  (DUTY  STATION,  DATES) 


ALL  OTHER  PREVIOUS  ASSIGNMENTS 


INTELLIGENCE  SCHOOLS  ATTENDED  (NMITC  BASIC,  NMITC  OPINTEL, 
ETC) _ 


FULL-TIME  WORK  EXPERIENCE  (IN  YEARS) _ 

HIGHEST  DEGREE  EARNED:  HIGH  SCHOOL _  ASSOCIATE'S _ 

BACHELOR'S _  GRADUATE _  OTHER  (SPECIFY) _ 

HOW  LONG  AGO  (IN  YEARS)  DID  YOU  COMPLETE  YOUR  HIGHEST  LEVEL  OF 
EDUCATION?  _ 

HOW  FAMILIAR  ARE  YOU  WITH  COMPUTERS,  GENERALLY? 

123456789 
Not  at  Very 

all  familiar  familiar 


HOW  MANY  HOURS  PER  WEEK  DO  YOU  USE  COMPUTERS? _ 

YOUR  GENERAL  COMMENTS  REGARDING  THE  EXPERIMENT: 
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TASK  DEBRIEF  SURVEY 


Cognitive  Feedback 


1.  How  would  you  rank  the  difficulty  of  your  task  during  the 
past  40  scenario  snapshots? 


1  2  3  4  5  6  7 

not 

at  all  difficult 


8  9 

very 

difficult 


2.  How  would  you  rank  the  clarity  of  your  task? 


1  2  3  4  5  6  7 

not 

at  all  clear 


8  9 

very 
clear 


3.  How  closely  does  the  task  resemble  what  you  do  on  a 
day-to-day  basis  in  the  fleet? 


1  2  3  4  5  6  7 

does  not 
resemble  at  all 


8  9 

resembl es 
very  closely 


4.  Have  you  performed  a  similar  task  in  the  past?  Y  N 
a.  If  "YES"  how  long  ago?  {in  years/months) _ 


5.  To  what  extent  were  the  concepts  of  decision  support 
information  clear  to  you  (as  explained  by  the  instructions  and 
the  experimenter)? 


1  2  3  4  5  6  7 

Not  clear 
at  all 


8  9 

Very 
clear 


6.  To  what  extent  was  decision  support  information  helpful  in 
improving  your  own  decision? 


1  23456789 

Not  at  all  Very 

helpful  helpful 

7.  To  what  extent  was  information  about  the  ship's  actual 
position  helpful  in  improving  your  own  decision? 

1  23456789 

Not  at  all  Very 

helpful  helpful 


56 


8.  Now  that  you  have  completed  the  task,  can  you  think  of  any 
other  factor  (other  variables,  etc)  that  may  have  influenced 
you  in  making  your  decision? 


9.  How  clear  were  the 
1  2  3  4 

Not  clear 
at  al  1 


instructions  regarding  the  task? 
5  6  7  8  9 


Very 

clear 


10.  How  would  you  present  the  task 

differently? _ 


11.  How  seriously  did  you  take  this  task,  generally? 


123456789 


not 

seriously 

12,  How  easy  was  this  system  to  use? 


very 

seriously 


12  3  4 

Not  at  all 
easy 


6  7 


8  9 

Very 
easy 
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TASK  DEBRIEF  SURVEY 

Outcome  Feedback 


1.  How  would  you  rank  the  difficulty  of  your  task  during  the 
past  40  scenario  snapshots? 


1  2  3  4  5  6  7 

not 

at  all  difficult 


8  9 

very 

dif f icul t 


2.  How  would  you  rank  the  clarity  of  your  task? 


1  2  3  4  5  6  7 

not 

at  all  clear 


8  9 

very 
clear 


3.  How  closely  does  the  task  resemble  what  you  do  on  a 
day-to-day  basis  in  the  fleet? 


1  2  3  4  5  6  7 

dees  not 
resemble  at  all 


8  9 

resembl es 
very  closely 


4.  Have  you  performed  a  similar  task  in  the  past?  Y  N 

a.  If  "YES"  how  long  ago?  (in  years/months) _ 

5.  Now  that  you  have  completed  the  task,  can  you  think  of  any 
other  factor  (other  variables,  etc)  that  may  have 
influenced  you  in  making  your  decision? 


6.  How  clear  were  the  instructions  regarding  the  task? 


1  2  3  4  5  6  7 

Not  clear 
at  all 


8  9 

Very 
clear 


7,  How  would  you  present  the  task 

differently? _ 
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8.  How  seriously  did  you  take  this  task,  generally? 


0  12 
not 

seriously 


3456789  10 

very 

seriously 


9.  To  what  extent  was  information  about  the  ship's  actual 


position  helpful  in  improving 

your 

own 

decision? 

1  2  3  4 

Not  at  all 
helpful 

5 

6 

7 

8  9 

Very 

helpful 

10.  How  easy  was  this 

system 

to  use? 

12  3  4 

Not  at  all 
easy 

5 

6 

7 

8  9 

Very 

easy 
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MAP  OF  CARIBBEAN 

(National  Geographic,  1988) 
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